DDL

Overview

info

This document is for TimeBase Community Edition and TimeBase Enterprise Edition 5.5.

The TimeBase Data Definition Language (DDL) allows users to create, drop, and modify streams in TimeBase by executing QQL statements, similar to CREATE TABLE ... in SQL.

In QQL, the equivalent statement begins with CREATE ... STREAM, and thus, is called the Create Stream statement.

Create a Stream

You can create two types of streams:

Syntax

The following grammar is used to create a stream:

CREATE TRANSIENT|DURABLE STREAM [IF NOT EXISTS] stream_name [title]
(class_expression|enum_expression [; ...])
[OPTIONS (identifier [= expr] [; ...])]
[COMMENT 'comment text']

class_expression is described as follows:

CLASS type_name [title] [UNDER type_name]
(static_attribute|attribute [, ...])
[AUXILIARY|NOT AUXILIARY]
[INSTANTIABLE|NOT INSTANTIABLE]
[COMMENT 'comment text']

enum_expression is described as follows:

ENUM enum_name [title]
(identifier [= expr] [, ...])
[FLAGS]
[COMMENT 'comment text']

static_attribute has the following grammar:

STATIC identifier [title] type [NOT NULL] [encoding] [BETWEEN min_expr AND max_expr] = expr
[TAGS (identifier:expr [, ...])]
[COMMENT 'comment text']

attribute has the following grammar:

identifier [title] type [NOT NULL] [encoding] [BETWEEN min_expr AND max_expr] [RELATIVE TO identifier] [DEFAULT expr]
[TAGS (identifier:expr [, ...])]
[COMMENT 'comment text']

Options

The OPTIONS list below contains all options for identifier:

Click to see stream `OPTIONS` attributes.

FIXEDTYPE: (Boolean.) A stream capable of containing messages of a single specified type.
POLYMORPHIC: (Boolean.) A polymorphic stream is capable of containing messages of several specified types.
LOSSLESS: (Boolean.) Lossless stream. Durable streams are always lossless.
LOSSY: (Boolean.) Lossy stream.
HIGHAVAILABILITY: (Boolean.) High availability durable streams are cached on startup.
PERIODICITY: (Varchar.) Indicate a known stream periodicity.
DF: (Numeric.) Distribution factor value.
INITSIZE: (Numeric.) Initial size of the write buffer in bytes.
MAXSIZE: (Numeric.) The limit on buffer growth in bytes. Default is 64K.
MAXTIME: (Numeric.) The limit on buffer growth as difference between first and last message time. Default is Long.MAX_VALUE.
UNIQUE: (Boolean.) Unique value.
STORAGEVERSION: (Varchar.) Stream format version. Supported versions are '5.0' (TS data file) and '4.3' (Classic).

Identifier

An identifier is a token that forms a name.

Tips

Use UNDER to indicate a parent class if applicable.
Classes may include static and/or non-static attributes. You can list them separated by a comma.
AUXILIARY classes cannot be written to a stream, but stream objects may contain AUXILIARY classes. NOT AUXILIARY classes can be written to a stream.
NOT INSTANTIABLE are abstract classes.
Use FLAGS to store enum values encoded as bitmask.
Supply a specific expr values with STATIC attributes.
Use BETWEEN/END to assign minimal and maximal values to numeric attributes.
Non-static attributes may acquire DEFAULT values.
Use RELATIVE TO to indicate that a non-static attribute's decoding depends on another field/property value.
With TAGS you can store a specific field-related metadata as a key:value pair.
Use COMMENT to add your message to the query.

For a general overview, refer to the QQL Introduction page.

Example

CREATE DURABLE STREAM TEST (
    CLASS "deltix.timebase.api.messages.MarketMessage" 'Market Message' (
        "currencyCode" 'Currency Code' INTEGER SIGNED (16) COMMENT 'Currency code represented as short',
        "originalTimestamp" 'Original Timestamp' TIMESTAMP COMMENT 'Exchange Time is measured in milliseconds that passed since January 1, 1970 UTC',
        "sequenceNumber" 'Sequence Number' INTEGER COMMENT 'Market specific identifier of the given event in a sequence of market events',
        "sourceId" 'Source Id' VARCHAR ALPHANUMERIC (10) COMMENT 'Identifies market data source' 
    ) AUXILIARY;
      
    CLASS "deltix.timebase.api.messages.BestBidOfferMessage" 'Quote Message' UNDER "deltix.timebase.api.messages.MarketMessage" (
        "offerPrice" 'Offer Price' FLOAT DECIMAL (2),
        "offerSize" 'Offer Size' FLOAT DECIMAL (0),
        "bidPrice" 'Bid Price' FLOAT DECIMAL (2) RELATIVE TO "offerPrice",
        "bidSize" 'Bid Size' FLOAT DECIMAL (0)
    );
    CLASS "deltix.timebase.api.messages.TradeMessage" 'Trade Message' UNDER "deltix.timebase.api.messages.MarketMessage" (
        "price" 'Trade Price' FLOAT DECIMAL (2),
        "size" 'Trade Size' FLOAT DECIMAL (0)
    );
)
OPTIONS (DF = 1; HIGHAVAILABILITY = FALSE)
COMMENT 'QQL is awesome'
/

tip

It is not allowed to create a stream with a duplicate key or name, so if you want to try and create a stream like bars, you have to change the key as well.

Modify a Stream

Syntax

The following grammar is used to modify a stream:

MODIFY STREAM stream_name [title]
(class_expr|enum_expr [; ...])
[OPTIONS (identifier [= expr] [; ...])]
[COMMENT 'comment text']
[CONFIRM NO_CONVERSION|CONVERT_DATA|DROP_ATTRIBUTES|DROP_TYPES|DROP_DATA]

class_expression is described as follows:

CLASS type_name [title] [UNDER type_name]
(static_attribute|attribute [, ...])
[AUXILIARY|NOT AUXILIARY]
[INSTANTIABLE|NOT INSTANTIABLE]
[COMMENT 'comment text']

enum_expression is described as follows:

ENUM enum_name [title]
(identifier [= expr] [, ...])
[FLAGS]
[COMMENT 'comment text']

static_attribute has the following grammar:

STATIC identifier [title] type [NOT NULL] [encoding] [BETWEEN min_expr AND max_expr] = expr
[TAGS (identifier:expr [, ...])]
[COMMENT 'comment text']

attribute has the following grammar:

identifier [title] type [NOT NULL] [encoding] [BETWEEN min_expr AND max_expr] [RELATIVE TO identifier] [DEFAULT expr]
[TAGS (identifier:expr [, ...])]
[COMMENT 'comment text']

Options

The OPTIONS list below contains all options for identifier:

Click to see stream `OPTIONS` attributes.

FIXEDTYPE: (Boolean.) A stream capable of containing messages of a single specified type.
POLYMORPHIC: (Boolean.) A polymorphic stream is capable of containing messages of several specified types.
LOSSLESS: (Boolean.) Lossless stream. Durable streams are always lossless.
LOSSY: (Boolean.) Lossy stream.
HIGHAVAILABILITY: (Boolean.) High availability durable streams are cached on startup.
PERIODICITY: (Varchar.) Indicate a known stream periodicity.
DF: (Numeric.) Distribution factor value.
INITSIZE: (Numeric.) Initial size of the write buffer in bytes.
MAXSIZE: (Numeric.) The limit on buffer growth in bytes. Default is 64K.
MAXTIME: (Numeric.) The limit on buffer growth as difference between first and last message time. Default is Long.MAX_VALUE.
UNIQUE: (Boolean.) Unique value.
STORAGEVERSION: (Varchar.) Stream format version. Supported versions are '5.0' (TS data file) and '4.3' (Classic).

Identifier

An identifier is a token that forms a name.

Confirm

The CONFIRM block indicates what operations are allowed when you want to change the schema:

NO_CONVERSION: Data convertion is not allowed and the MODIFY statement will fail (for example, when data type of a field changes from INTEGER to FLOAT).
CONVERT_DATA: Any data convertion is allowed.
DROP_ATTRIBUTES: Removing fields is allowed.
DROP_TYPES: Removing types is allowed.
DROP_DATA: Allow removing data that cannot be converted (for example, when the data type changes from VARCHAR to FLOAT).

Tips

Use UNDER to indicate a parent class if applicable.
Classes may include static and/or non-static attributes. You can list them separated by a comma.
AUXILIARY classes cannot be written to a stream, but stream objects may contain AUXILIARY classes. NOT AUXILIARY classes can be written to a stream.
NOT INSTANTIABLE are abstract classes.
Use FLAGS to store enum values encoded as bitmask.
Supply a specific expr values with STATIC attributes.
Use BETWEEN/END to assign minimal and maximal values to numeric attributes.
Non-static attributes may acquire DEFAULT values.
Use RELATIVE TO to indicate that a non-static attribute's decoding depends on another field/property value.
With TAGS you can store a specific field-related metadata as a key:value pair.
Use COMMENT to add your message to the query.

For a general overview, refer to the QQL Introduction page.

Example

MODIFY STREAM TEST (
    CLASS "deltix.timebase.api.messages.MarketMessage" 'Market Message' (
        "currencyCode" 'Currency Code' INTEGER SIGNED (16) COMMENT 'Currency code represented as short',
        "originalTimestamp" 'Original Timestamp' TIMESTAMP COMMENT 'Exchange Time is measured in milliseconds that passed since January 1, 1970 UTC',
        "sequenceNumber" 'Sequence Number' INTEGER COMMENT 'Market specific identifier of the given event in a sequence of market events',
        "sourceId" 'Source Id' VARCHAR ALPHANUMERIC (10) COMMENT 'Identifies market data source' 
    ) AUXILIARY;
      
    CLASS "deltix.timebase.api.messages.BestBidOfferMessage" 'Quote Message' UNDER "deltix.timebase.api.messages.MarketMessage" (
        "offerPrice" 'Offer Price' FLOAT DECIMAL (2),
        "offerSize" 'Offer Size' FLOAT DECIMAL (0),
        "bidPrice" 'Bid Price' FLOAT DECIMAL (2) RELATIVE TO "offerPrice",
        "bidSize" 'Bid Size' FLOAT DECIMAL (0)
        "exchangeId" 'Exchange' VARCHAR ALPHANUMERIC (10) COMMENT 'Exchange Code',
    );
    CLASS "deltix.timebase.api.messages.TradeMessage" 'Trade Message' UNDER "deltix.timebase.api.messages.MarketMessage" (
        "price" 'Trade Price' FLOAT DECIMAL (2),
        "size" 'Trade Size' FLOAT DECIMAL (0)
    );
)
OPTIONS (DF = 1; HIGHAVAILABILITY = FALSE)
COMMENT 'QQL is awesome'
/

Alter a Stream

info

The ALTER STREAM statement is available starting with TimeBase Enterprise Edition 5.6.67+

In contrast to the MODIFY statement that necessitates specifying the entire schema for any alteration, the ALTER STREAM statement introduces the capability to perform discrete modifications.

This distinction allows for a more precise and targeted approach when altering specific aspects of a stream's schema, ensuring that the changes are localized to the intended attributes or elements without affecting the entire structure.

Alter Stream Statement

At a high-level, the ALTER STREAM statement is structured as follows:

ALTER STREAM stream_name (
    alter class|enum |
    add class|enum |
    drop class|enum |
    rewrite class|enum
    [; ...]
) 
[SET stream_option_name [=] value [, SET ...]]
[CONFIRM NO_CONVERSION|CONVERT_DATA|DROP_ATTRIBUTES|DROP_TYPES|DROP_DATA]

ALTER STREAM can contain one or more alter class and alter enum expressions, each describing modifications of the class.

When using ALTER STREAM, you can add any number of the following changes:

alter
add
drop
rewrite

Options

Click to see the available options in stream_option_name.

KEY: (Varchar.) Stream key.
NAME: (Varchar.) Stream name.
DESCRIPTION: (Varchar.) Stream description.
OWNER: (Varchar.) Stream owner.
DF: (Numeric.) Distribution factor value.
PERIODICITY: (Varchar.) Indicate a known stream periodicity.
INITSIZE: (Numeric.) Initial size of the write buffer in bytes.
MAXSIZE: (Numeric.) The limit on buffer growth in bytes. Default is 64K.
MAXTIME: (Numeric.) The limit on buffer growth as difference between first and last message time. Default is Long.MAX_VALUE.
LOSSY: (Boolean.) Lossy stream.
LOSSLESS: (Boolean.) Lossless stream. Durable streams are always lossless.
HA: (Boolean.) High availability durable streams are cached on startup.

Example

Here is an example of changing stream options:

ALTER STREAM bars
  SET PERIODICITY '1D',
  SET OWNER 'admin',
  SET DESCRIPTION '1 day bars'

Alter Class Expression

At the top level, the ALTER CLASS statement is similar to the ALTER STREAM statement:

ALTER CLASS type_name (
    alter field |
    add field |
    drop field |
    rewrite field 
    [; ...]
)
[SET class_option_name [=] value [, SET ...]]
[RESOLVE identifier DEFAULT expr]

However, instead of the ALTER CLASS expression, it can contain the ALTER FIELD expression with its own list of options.

Additionally, the ALTER CLASS statement contains a RESOLVE section that can be specified to dictate how alterations should be addressed in scenarios where a field lacks a default value.

Options

Click to see the available options in class_option_name.

NAME: (Varchar.) Class name.
TITLE: (Varchar.) Class title.
DESCRIPTION: (Varchar.) Class description.
UNDER: (Varchar.) Parent class name.
AUXILIARY: (Boolean.) Specify for auxilliary classes.
INSTANTIABLE: (Boolean.) Specify for instantiable classes.

Example

Here's an example of renaming type:

ALTER STREAM KRAKEN 
  ALTER CLASS "deltix.qsrv.hf.plugins.data.kraken.types.KrakenPackageHeader"
    SET NAME "deltix.timebase.api.messages.universal.CustomPackageHeader",
    SET DESCRIPTION 'Custom Package Header', 
    SET TITLE 'Custom Package Header'

Alter Field Expression

The ALTER FIELD expression allows you to change one or more field options:

ALTER FIELD field_name 
SET field_option_name [=] value [, SET ...]
[DEFAULT expr]

The DEFAULT section provides a mechanism to assign a default value to be able to resolve values for NOT NULL fields.

Options

Click to see the available options in field_option_name.

NAME: (Varchar.) Field name.
TITLE: (Varchar.) class title.
DESCRIPTION: (Varchar.) class description.
TYPE: (Varchar.) Field type.
ENCODING: (Varchar.) Field encoding.
NULL, NOT NULL: (Boolean.) Alter field nullability.
STATIC, NOT STATIC: (Boolean.) Make field static/not static.
MIN: Field min value.
MAX: Field min value.
RELATIVE TO: Set field relative to another field.
TAGS: Set field attributes.

Example

Here's an example of example of changing the originalTimestamp field encoding to nanoseconds:

ALTER STREAM kraken 
  ALTER CLASS "deltix.timebase.api.messages.MarketMessage" 
    ALTER FIELD "originalTimestamp" SET ENCODING NANOSECOND
CONFIRM CONVERT_DATA

Add/Rewrite Field Expression

The ADD FIELD and REWRITE FIELD expressions are designed to fully describe the final structure of a field following its creation or modification, respectively.

When utilizing an ADD FIELD expression, you are providing a complete definition of a new field to be added to the schema, detailing its type, constraints, and any other necessary attributes.

The REWRITE FIELD expression is used for redifining existing fields.

The syntax of the ADD FIELD and REWRITE FIELD expressions matches the syntax used for these expressions in the CREATE and MODIFY statements.

Add/Rewrite Non-Static Field

The syntax for adding or rewriting a non-static field is as follows:

ADD|REWRITE FIELD identifier [title] type [NOT NULL] [encoding] [BETWEEN min_expr AND max_expr] [RELATIVE TO identifier] [DEFAULT expr]
[TAGS (identifier:expr [, ...])]
[COMMENT 'comment text']

Add/Rewrite Static Field

The syntax for adding or rewriting a static field is as follows:

ADD|REWRITE FIELD STATIC identifier [title] type [NOT NULL] [encoding] [BETWEEN min_expr AND max_expr] = expr
[TAGS (identifier:expr [, ...])]
[COMMENT 'comment text']

Example

Here's an example of applying several changes to several types:

ALTER STREAM KRAKEN (
  ALTER CLASS "deltix.timebase.api.messages.universal.TradeEntry" (
    ALTER FIELD "buyerNumberOfOrders" 
      SET STATIC 1, 
      SET TITLE 'Static Number Of Orders';
    ALTER FIELD "buyerOrderId" 
      SET TYPE INTEGER INT64,
      SET TITLE 'Integer Order ID'
  );
  ALTER CLASS "deltix.qsrv.hf.plugins.data.kraken.types.KrakenPackageHeader" (
    ADD FIELD "packageReceiveTime" TIMESTAMP NANOSECOND;
  ) 
  SET DESCRIPTION 'Package Header with receive timestamp', 
  SET TITLE 'Kraken Package Header';
)
CONFIRM DROP_DATA

Drop Field Expression

DROP FIELD field_name

Alter Enum Expression

The ALTER ENUM expression provides the functionality to add, remove or modify the content of an enum type. For example:

# Alter enum syntax
ALTER ENUM enum_name (
    ALTER identifier SET VALUE [=] expr |
    ALTER identifier1 SET NAME [=] identifier2 |
    RENAME identifier1 TO identifier2 | # alias for ALTER ... SET NAME
    ADD identifier = expr |
    DROP identifier |
    REWRITE identifier = expr
    [; ...]
)

This includes the ability to:

Add new values to the enum
Remove existing values
Alter the names and associated values

Example

Here's an example of adding an enum value:

ALTER STREAM KRAKEN 
  ALTER ENUM "deltix.timebase.api.messages.service.DataConnectorStatus"
      ADD "DESCONNECTED_BY_VENDOR" = 11
CONFIRM CONVERT_DATA

Add/Rewrite Class/Enum Expression

The ADD EXPRESSION and REWRITE EXPRESSION within the ALTER STREAM statement are used to define the complete structure of a class or enum as it will appear post-creation or post-modification, respectively.

The syntax for these expressions is consistent with that of the CREATE and MODIFY statements.

This means that when you use an ADD FIELD or REWRITE EXPRESSION, you are specifying the full definition of an element.

Add/Rewrite Class

The add class and rewrite class syntax looks as follows:

ADD|REWRITE CLASS type_name [title] [UNDER type_name]
(static_attribute|attribute [, ...])
[AUXILIARY|NOT AUXILIARY]
[INSTANTIABLE|NOT INSTANTIABLE]
[COMMENT 'comment text']

static_attribute has the following grammar:

STATIC identifier [title] type [NOT NULL] [encoding] [BETWEEN min_expr AND max_expr] = expr
[TAGS (identifier:expr [, ...])]
[COMMENT 'comment text']

attribute has the following grammar:

identifier [title] type [NOT NULL] [encoding] [BETWEEN min_expr AND max_expr] [RELATIVE TO identifier] [DEFAULT expr]
[TAGS (identifier:expr [, ...])]
[COMMENT 'comment text']

Add/Rewrite Enum

The add enum and rewrite enum syntax looks as follows:

ADD|REWRITE ENUM enum_name [title]
(identifier [= expr] [, ...])
[FLAGS]
[COMMENT 'comment text']

Example

The following example:

Adds a new enum deltix.timebase.api.messages.service.ResetReason
Changes the class deltix.timebase.api.messages.universal.BookResetEntry by adding a field to the class that points to the new enum.

ALTER STREAM KRAKEN (
  ADD ENUM "deltix.timebase.api.messages.service.ResetReason" (
    "UNKNOWN" = 0,
    "DISCONNECTED" = 1
  );
  ALTER CLASS "deltix.timebase.api.messages.universal.BookResetEntry" (
    ADD FIELD "reason" "deltix.timebase.api.messages.service.ResetReason"
  );
)

You can add the L3Entry class and rewrite the type of entries to be able to use it as a part of a polymophic array:

ALTER STREAM KRAKEN (
    ADD CLASS "deltix.timebase.api.messages.universal.L3Entry" UNDER "deltix.timebase.api.messages.universal.BasePriceEntry" (
      "vendorQuoteId" VARCHAR NOT NULL
    ) AUXILIARY;
    ALTER CLASS "deltix.qsrv.hf.plugins.data.kraken.types.KrakenPackageHeader" (
      REWRITE FIELD entries ARRAY(OBJECT(
          "deltix.timebase.api.messages.universal.L1Entry", 
          "deltix.timebase.api.messages.universal.L2EntryNew", 
          "deltix.timebase.api.messages.universal.L2EntryUpdate", 
          "deltix.qsrv.hf.plugins.data.kraken.types.KrakenTradeEntry", 
          "deltix.timebase.api.messages.universal.BookResetEntry",
          "deltix.timebase.api.messages.universal.L3Entry") NOT NULL) NOT NULL
    )
)

Drop Class/Enum Expression

DROP CLASS|ENUM type_name

Drop a Stream

The following grammar is used:

DROP STREAM [IF EXISTS] stream_name

For example:

DROP STREAM test

tip

You will notice that the capitalization of the stream key is not important. That is because, in all cases, QQL searches for existing identifiers in a case-insensitive manner, as long as they are unambiguous.

Overview​

info

Create a Stream​

Syntax​

Options​

Identifier​

Tips​

Example​

tip

Modify a Stream​

Syntax​

Options​

Identifier​

Confirm​

Tips​

Example​

Alter a Stream​

info

Alter Stream Statement​

Options​

Example​

Alter Class Expression​

Options​

Example​

Alter Field Expression​

Options​

Example​

Add/Rewrite Field Expression​

Add/Rewrite Non-Static Field​

Add/Rewrite Static Field​

Example​

Drop Field Expression​

Alter Enum Expression​

Example​

Add/Rewrite Class/Enum Expression​

Add/Rewrite Class​

Add/Rewrite Enum​

Example​

Drop Class/Enum Expression​

Drop a Stream​

tip

Overview

Create a Stream

Syntax

Options

Identifier

Tips

Example

Modify a Stream

Syntax

Options

Identifier

Confirm

Tips

Example

Alter a Stream

Alter Stream Statement

Options

Example

Alter Class Expression

Options

Example

Alter Field Expression

Options

Example

Add/Rewrite Field Expression

Add/Rewrite Non-Static Field

Add/Rewrite Static Field

Example

Drop Field Expression

Alter Enum Expression

Example

Add/Rewrite Class/Enum Expression

Add/Rewrite Class

Add/Rewrite Enum

Example

Drop Class/Enum Expression

Drop a Stream